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Abstract 


This chapter describes how the different ICARUS unmanned search and rescue tools 
have been evaluated and validated using operational benchmarking techniques. Two 
large-scale simulated disaster scenarios were organized: a simulated shipwreck and 
an earthquake response scenario. Next to these simulated response scenarios, where 
ICARUS tools were deployed in tight interaction with real end users, ICARUS tools also 
participated to a real relief, embedded in a team of end users for a flood response mission. 
These validation trials allow us to conclude that the ICARUS tools fulfil the user require- 
ments and goals set up at the beginning of the project. 


Keywords: rescue robotics, operational validation 


1. Introduction 


As shown in the previous chapters of this book, the ICARUS project developed multiple 
unmanned systems and tools for supporting search and rescue (SAR) teams. These technolog- 
ical tools were developed after a careful consideration of the end-user needs [1], as discussed 
in Chapter 2 of this book. Of course, during and at the end of the design lifecycle, the per- 
formance of the different tools with respect to the user requirements and target performance 
levels needs to be evaluated. This process of system validation requires a careful compromise 
between two points of view: 
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1. End users want validation tests to occur in realistic operational conditions, mimicking as 
closely as possible a real deployment. 


2. Scientists want validation tests to have statistical relevance, so they want repeated tests, 
performed under controlled environments. However, it is very hard to quantify the system 
performance in a rigorous scientific manner due to the fact that many variables are out of 
control in an outdoor environment, e.g. the weather conditions (wind, rain, sea state, illu- 
minance, etc.). Moreover, a scientific evaluation requires that multiple trials must be held 
to validate the statistical significance of the quantitative results, which is not evident when 
confronted with the evaluation of complex heterogeneous robotic teams in operational 
conditions, requiring significant logistics for setting up each trial run. 


In the past, multiple proposals have been made in order to combine these different points 
of view [2]. As a result, validation methodologies can be generally categorized into two 
approaches: 


1. The first approach is based on the development of highly standardized test methodologies 
[3]. A good example is that developed and proposed by the National Institute of Standards 
and Technology (NIST). The big advantage of these methodologies is that they allow to 
accurately quantify the performance of the robots in a number of test setups. The disad- 
vantage of these methods is that, due to their standardized nature, these approaches are 
often quite dissociated from practical operational conditions. 


2. The second approach for validation is robot competitions like DARPA [4], euRathlon [5] 
and ELROB [6]. Here, multiple robotic systems are pitted against each other in more or less 
realistic operating conditions. The advantage of this validation approach is that the perfor- 
mance can be evaluated in real-life like circumstances and environments. The disadvan- 
tage of these kinds of benchmarking methodologies is that, due to their non-standardized 
nature, they often only allow a qualitative appreciation of the robot performance and do 
not allow making a detailed quantitative measurement. Another important disadvantage 
is that coincidence (e.g. dependence on singular element failures that may not be exemplar 
for the overall system operation, changing weather and lighting conditions between trial 
runs, etc.) plays an important role in these competitions, which significantly compromises 
the statistical significance of the benchmarking result. 


It is clear that both of these approaches are highly valuable and necessary. However, none of 
them gives an ultimate solution for the performance evaluation problem. Here, we present the 
operational test and validation approach for the evaluation of the performance of a range of 
marine, aerial and ground search and rescue robots. This methodology has been proposed and 
followed within the ICARUS project. The proposed approach aims to find a compromise between 
the traditional rigorous standardized approaches and the more open-ended robot competitions. 
Following this methodology, operational scenarios are defined that include a performance 
assessment of individual robotic tools. Furthermore, these operational scenarios also assess the 
performance of heterogeneous teams of robotic tools, cooperating not only among robots, but 
also with manned teams in realistic search and rescue activities. In this way, it is possible to per- 
form a more complete validation of the use of robotic tools in challenging real-world scenarios. 
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The ICARUS project considers two main demonstrations to validate the tools developed dur- 
ing the whole duration of the project [7]: 


1. A marine demonstration, simulating a shipwreck in coastal waters. 


2. A land demonstration, simulating an earthquake in an urban environment. 


This chapter reports on the results of the operational validation performed during both dem- 
onstrations. However, the proof of the pudding is in the eating. No simulated disaster man- 
agement exercise can mimic the chaos and difficult environmental conditions encountered 
during a real response operation. Therefore, we also included a section reporting on the use 
of one of the unmanned aerial systems deployed during a real flood-relief operation, within 
the framework of the ICARUS project. 


2. Maritime demonstration: simulated shipwreck response 


2.1. General storyboard 


Organized in coordination with the yearly REX exercises [8], the scenario for the maritime dem- 
onstration is based on a shipwreck of a ferryboat. For such a scenario, the roles of the robotic 
tools can be separated according to the nature of the platforms. Aerial vehicles are used in 
search operations, sweeping the area and providing information about the exact location of the 
accident, localizing victims on the water and tracking them. Another role of the aerial segment is 
to carry mobile communication equipment allowing for the establishment of an extended range 
mobile network to support communications between all robotic assets and also with manned 
platforms. On the other hand, the main role of maritime platforms is on the rescue operations. 
Taking advantage of the data collected by the aerial segment, maritime platforms get close to 
located victims and assist them. Such assistance consists in providing floatation and shelter from 
environmental conditions, extending their time of life and allowing rescue in safe conditions. 


2.2. Location and organisational issues 


The sea demonstration took place on the Tagus river estuary in the area of the Lisboa Naval 
Base, located in Alfeite, Almada, Portugal, as shown in Figure 1. The selection of this place 
took into account several issues related to the demonstration: segregated area for the opera- 
tion of unmanned systems, realistic scenario for a search and rescue operation, easy access to 
the operational area and for mounting command and control stations as well as communica- 
tions equipment. 


Figure 1 shows the location of the ICARUS sea demonstration area, off the south bank of 
Tagus river estuary. This figure also exhibits the navigation lanes of ferryboats that continu- 
ously cross the estuary, a characteristic that was taken into account in the selection of a realis- 
tic location for the demonstration. Furthermore, the selection of a location right in front of the 
Lisboa Naval Base simplified the logistics associated to the operation due to the existence of 
local facilities from the Portuguese Navy and from the Arsenal do Alfeite Shipyard. 
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ICARUS Sea Demonstration 


Figure 1. Location of the sea demonstration area: Lisboa Naval Base in Portugal, showcasing the area reserved for the 
ICARUS sea demonstration (source: ICARUS). 


For the operations on the water, the region shown in Figure 1 was closed to maritime traffic, 
except for the vessels required for the demonstration and to patrol the area to make sure that 
unauthorized vessels do not enter it, so that all activities related to the demonstration could be 
carried out without major concerns with other activities that could cause any kind of interfer- 
ence. This area exceeds 2 km? and the maximum distance from shore is about 2.2 km. 


2.3. Operational validation for maritime search and rescue 


Time: TO 


Events: An explosion of unknown origin occurs in a ferryboat crossing the Tagus river estuary 
near Lisboa. Victims fall in the water. The ferryboat, shown in Figure 2, starts sinking. 


Figure 2. Ferryboat used for simulating the shipwreck accident (source: ICARUS). 
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Time: TO + OhO5min 


Events: The Marine Rescue Coordination Center (MRCC) in Lisboa receives an alert describ- 
ing the accident and its approximate location. After an initial assessment, the MRCC dis- 
patches to the area search and rescue teams (including robotic assets). 


Time: TO + 0h40min 


Events: Search and rescue teams start arriving at the location of the accident (riverbank). 
A local coordination centre, shown in Figure 3, is set up where the ICARUS command and 
control system [9] is deployed and communications with MRCC are established. Search and 
rescue assets, including manned rigid-hulled inflatable boats and unmanned aerial and sur- 
face vehicles, are prepared for launching. 


Evaluation: For a search and rescue mission where every minute counts, the set-up time for 
the developed technological tools is still an important factor where progress can be made. 
Confronted with the complicated frequency spectrum in the Lisbon harbour environment, it was 
mainly the configuration of the communication tools which increased the overall set-up time. 


Figure 3. Set-up of ICARUS command and control tools at the local coordination centre. (source: ICARUS). 


Time: TO + 1h00min 


Events: An area around the approximate location of the accident is defined and a fixed wing 
long endurance unmanned aircraft is deployed to make a survey of that area, as shown in 
Figure 4. 


Evaluation: Take-off and landing operations of the fixed wing long endurance unmanned 
aircraft were performed manually by trained personnel. As the fixed wing long endur- 
ance unmanned aircraft is hand-launched, take-off is still quite easy, even in a cluttered 
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environment. However, landing requires the careful choice of a suited landing position, which 
is tough in a heavily built-up harbour environment. Luckily, the fixed wing long endurance 
unmanned aircraft can stay airborne for multiple hours (even days), so the issue does not 
occur often. In flight, the fixed wing long endurance unmanned aircraft semi-autonomously 
executed a GPS-defined trajectory and a search pattern mission over the area defined for 
the sea demonstration. A person external to the operating team provided the profile of this 
mission. Several flight patterns were tested and successfully performed by the fixed-wing 
long-endurance unmanned aircraft. 


Figure 4. Initial area surveillance with the fixed wing long endurance unmanned aircraft (source: ICARUS). 


Time: TO + 1h10min 


Events: Images collected by the fixed wing long endurance unmanned aircraft start arriving 
at the local coordination centre and at the MRCC, providing information about the location of 
the ferryboat, victims on the water and debris scattered over the area, as shown on Figure 5. 


Evaluation: Live streams from the thermal and video camera on-board the fixed wing long 
endurance unmanned aircraft were received at the command and control interface. Victim 
positions were also transferred to the command station. 


Time: TO + 1h20min 


Events: Based on the information collected by the fixed wing long endurance unmanned 
aircraft, the rescue operation is planned at the MRCC: areas of intervention are assigned to 
manned rigid-hulled inflatable boats and carrier unmanned surface vehicles. The MRCC 
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sends operation plans to local coordination centre; these plans are received by the command 
and control station of robotic assets and rescue operations start. 


Evaluation: In order to test the detection capabilities of the different assets, the victims in 
the water were asynchronously spread over multiple clusters: there were clusters with one 
person, two persons and four persons. The relatively high-altitude fixed wing long endur- 
ance unmanned aircraft proved to be able to spot the clusters of victims in the water, but was 


Figure 5. Information being displayed on the command and control interface (source: ICARUS). 


unable to count the number of victims per cluster, due to limitations in sensor resolution and 
the high flight altitude. 


Time: T0 + 1h30min 


Events: Carrier unmanned surface vehicles head to clusters of victims located further away; 
at the same time, the rotary wing unmanned aircraft is launched to track victims that will be 
rescued by unmanned capsules on-board two carrier unmanned surface vehicles: the ROAZ 
II and the U-RANGER, as shown on Figure 6. Manned rigid-hulled inflatable boats also 
depart to rescue people closer to the margin, while another unmanned capsule is deployed 
and remotely piloted towards victims that are close to the margin. 
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Evaluation: Similar to the endurance aircraft, the take-off and landing of the rotary wing 
unmanned aircraft were performed manually and mapping and victim search were exe- 
cuted semi-autonomously. For legal and safety reasons, the U-RANGER always operated 
with a person on-board. Besides the command and control interface that remotely operated 
the unmanned surface vehicle, a second backup control station was used to create another 
safety loop. Contrary to the fixed-wing long-endurance unmanned aircraft, the rotary-wing 
unmanned aircraft proved to be capable of not only detecting the clusters of victims but also 
of counting the number of victims per cluster, which provided important information for 
the allocation of resources. Space management between manned and unmanned assets was 
an important factor in this phase of validation, as multiple simultaneous rescue operations 
started in the same area. This incurred that the U-RANGER could not operate at full speed due 
to safety reasons, but due to the intelligent obstacle avoidance capabilities of the unmanned 
systems, no problems occurred. 


Figure 6. Rotary wing unmanned aircraft providing victim’s location to ROAZ II (source: ICARUS). 


Time: TO + 1h40min 


Events: One carrier unmanned surface vehicle arrives near a first cluster of victims and 
deploys one unmanned capsule, as shown on Figure 8. This unmanned capsule is remotely 
operated to move towards the victims using location information provided by the rotary- 
wing unmanned aircraft, as shown on Figure 7. When the unmanned capsule arrives close 
to the victims, as shown on Figure 9, it inflates a life raft and victims start climbing on-board 
the life raft. 
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Figure 7. Rotary wing unmanned aircraft providing victim location to an unmanned capsule (source: ICARUS). 


Evaluation: Once started, the rescue operations moved very quickly. During the whole opera- 
tion, live video feeds from thermal and visible cameras were received at the base station con- 
sole. Radar and laser data used to detect obstacles on the water were also transmitted to shore. 


Time: T0 + 1h45min 


Events: The same carrier unmanned surface vehicle moves now towards another cluster of 
victims. The rotary wing unmanned aircraft tracks the location of these victims that are drift- 
ing away due to the water current. Another unmanned capsule is launched and automatically 
moves towards the victims using information provided by the rotary wing unmanned aircraft. 
Again, it inflates the life raft when close to the victims. Meanwhile, the other unmanned sur- 
face vehicle also deploys an unmanned capsule; this unmanned capsule moves autonomously 
towards a location where victims were spotted; the life raft is inflated by direct action of the 
victims. By that time, the unmanned capsule launched from the riverbank already reached the 
victims and its life raft is inflated. 


Figure 8. Unmanned capsule being launched from ROAZ II (source: ICARUS). 
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Figure 9. Victim being rescued by an unmanned capsule (source: ICARUS). 


Evaluation: Simultaneous rescue operations normally provide a cognitive overload for the 
commander in charge of the operation, but thanks to the ecologic display functionalities of the 
ICARUS command and control system, the commander could keep an overview of the differ- 
ent operations and coordinate the instructions towards the different team members. 


Time: TO + 1h50min 


Events: The carrier unmanned surface vehicles head now to the area behind the sinking ferry- 
boat, as shown on Figure 10; thermal and visible images are sent back to the local coordination 


Figure 10. U-RANGER and ROAZ II providing situational awareness to rescue workers behind the ferryboat (source: 
ICARUS). 
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centre for situation assessment. The aerial vehicles provide wireless links to the unmanned 
surface vehicles while they are operating out of sight. 


Evaluation: The ferryboat blocked line of sight connectivity to the unmanned surface vehicles. 
However, the fixed wing long endurance unmanned aircraft acted as a relay station, provid- 
ing connectivity to both vehicles, allowing for streaming real-time video to the base station. 


Time: T0 + 2h00min 


Events: Information received from the unmanned surface vehicles is then used to plan a res- 
cue operation for people still on the ferryboat. By that time, the robotic assets are recovered 
and their activities end. 


2.4. Conclusions of the marine demonstration 


Besides the impact of the sea scenario demonstration in the media and the opportunity it pro- 
vided for establishing contacts with stakeholders and other relevant players, the experiments 
conducted during the trials were used to obtain qualitative and quantitative information 
about the technical developments of the project and the way they met the established goals. 


Such assessment was organized along the following lines: 


1. Validation of capabilities. During the experiments, capabilities (sets of aggregated require- 
ments) planned for each system or set of systems were assessed. A total of 71 of 84 capabili- 
ties were validated. 


2. Performance analysis. Performance analysis consisted in assessing quantitative metrics 
against three performance targets (minimum acceptance, goal and breakthrough levels). 
For the total of 66 metrics considered, 36 (55%) reached the breakthrough level, 18 (27%) 
reached the goal level, 11 (17%) reached the minimum acceptance level and just 1 (2%) was 
below this minimum level. 


Two officers from the Portuguese Navy were asked to evaluate the individual platform exper- 
iments, either when they were conducted independently or while they were carried out as 
part of more complex tests. In each case, they were asked to score the experiment and provide 
some comments. The feedback provided were extremely positive and in line with the other 
evaluation methods. 


3. Land demonstration: simulated earthquake response 


3.1. General storyboard and setting 


In order to validate the performance of the ICARUS tools in an urban search and rescue con- 
text, the ICARUS land demonstration defines an earthquake-response scenario where the dif- 
ferent ICARUS aerial and ground assets are used to support the relief teams. The ICARUS 
land demonstration was integrated into a training exercise of the Belgian First Aid and 
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Support Team (B-FAST) as a preparation for their INSARAG IEC re-classification tests. As 
such, the complete integration of unmanned tools in the standard operating procedures of 
real search and rescue workers could be tested. 


The ICARUS land validation took place in the military base Camp Roi Albert in Marche- 
en-Famenne, Belgium, a woody and hilly area halfway between Brussels and Luxemburg. The 
base is the regular training ground for the B-FAST team and provides for this purpose a rubble 
field with a pancake house for performing victim search and rescue operations and a built-up 
area which can serve as a mock-up urban setting for testing urban search and rescue protocols. 


Figure 11 shows the different areas within the simulated crisis area which were used through- 
out the operations: 


e The Base of Operations (BoO) set up by the B-FAST team. 

e The location of the city of Focagne, which is the urban area assigned to the B-FAST team. 
e The Forward BoO which is set up by B-FAST close to the Focagne city. 

e The road from the BoO to the Forward BoO which is partially blocked by debris. 


e Apartment buildings which have collapsed and where victims could be found in the voids 
between the rubble. 


e Asemi-demolished school building where trapped school children are present. 
e A warehouse on fire with chemical products inside. 


In the rest of this section, we will explain how the different ICARUS tools assist the search and 
rescue workers in dealing with each of the presented difficulties. 


Ghremical/warehouse on fire 
Demolished School Building 


Cityyofikocagne 


FojwardiBoOy 


Demolished’Appartment Blocks 


Blocked Roa \ 


D 


——— 
W ee BAaSeof Cee ea 


Figure 11. Situational overview of the crisis area (source: ICARUS). 
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3.2. Integration of the ICARUS system into the OSOCC 


Events: As shown on Figure 12, the B-FAST Urban Search and Rescue (USAR) team 
sets up the On-Site Operations Coordination Centre (OSOCC)-level C4I equipment 
which includes a large workstation with displays. The USAR team’s communication specialist 
sets up local communication equipment and tests to see if web access and local GSM net- 
works are available. The ICARUS C4I systems then connect to the Global Disaster Alert and 
Communication System (GDACS) and pulls in the latest data about the disaster. 


Figure 12. OSOCC set up by the B-FAST team, serving as base command station for the operations (source: ICARUS). 


Evaluation: The ICARUS command and control system was successfully connected to the 
OSOCC and imported GIS data, fact sheet data and GDACS data about the disaster. The end 
users did evaluate the total setup time of the whole system (1 hour) as still too slow. The main 
issue is communication, which is of course not an easy parameter to quantify and debug, as 
the communication ecosystem and frequency spectrum usage will be different in every crisis. 


3.3. Support to mission planning 


Events: As very little information is available, the planning officials of the team decide to use 
unmanned tools to obtain quickly a better common operational picture of the situation. The 
B-FAST team leader orders the fixed wing endurance aircraft to scan the city of Focagne, as 
shown on Figure 13. The human operator at the BoO selects a geo-referenced scan on the 
human-machine interface, after which the aircraft executes the task semi-autonomously. 


Figure 13. Launching the endurance aircraft during the demonstration (source: ICARUS). 
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The goal of this first scan is to do obtain a good overview of the level of destruction in 
Focagne. 


Evaluation: The UAS autonomously acquired data (visual + IR imaging) over the area of 
interest and transmitted this data in real time to the base station. The data from the UAS were 
used to reconstruct a map, which the mission planner overlapped with the pre-existing GIS 
data. Based on the obtained information, the mission planner used the command and control 
system for sectorization. More detailed scans per sector were then performed. 


End users were very impressed with the speed of obtaining a high-quality situational over- 
view of the crisis area using the data gathered by the aircraft via the ICARUS command and 
control system and many positive remarks were voiced. 


3.4. Deployment of the USAR team 


Events: As shown on Figure 14, the USAR teams move towards and deploy into a sector 
assigned by the mission planner via the command and control system. The main purpose 
of this scenario is to test the (rapid) deployment capabilities and the integration of the com- 
munication and command and control system. Another purpose of this scenario is to test 
the network and command and control system management capabilities when confronted 
with dynamic team and resource allocations and to test the capability of the aircraft to detect 
roadblocks. 


Figure 14. B-FAST team advancing with all ICARUS tools: large UGV (driving in front), small UGV (packed on first 
vehicle), rotorcraft (airborne in the middle of the picture) and fixed wing aircraft (airborne, but not visible in this picture) 
(source: ICARUS). 


Evaluation: The B-FAST team moved from the base of operations towards the forward base of 
operations, together with all the ICARUS tools. Organising this scenario required convincing 
the end users of the added value the ICARUS tools could bring in this phase of the operation. 
Indeed, in the beginning, the end users were afraid that the ICARUS tools would needlessly 
delay the USAR deployment operation, whereas speed is a key issue of course in any search 
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and rescue operation. For this deployment operation, mostly the large UGV posed issues as it 
is of course not as fast as a standard truck. The ICARUS team therefore worked hard on find- 
ing the right balance between teleoperation and autonomous guidance for driving the large 
UGV as fast as possible up the hill without slowing down the B-FAST convoy. This succeeded 
very well, as the convoy could advance at a very normal speed during the public demonstra- 
tion day, to the satisfaction of the B-FAST users. The B-FAST end users also highly appreci- 
ated the continuous live input from the outdoor rotorcraft, warning them about road blocks, 
which could save them valuable time in a real operation. 


3.5. Victim search and rescue in demolished apartment buildings 


Events: The USAR team rescues victims trapped in a semi-demolished apartment building, 
helped by the ICARUS UGV and UAV systems. The main objective of this scenario is to test 
the assessment, search and rescue capabilities of the outdoor rotorcraft and the large UGV 
and their collaborative operation mode. 


The fixed wing aircraft is sent to sector to perform long-range human detection using its 
infrared detector. It scans the area where the apartment buildings have collapsed. The UAS 
returned a map indicating the locations of potential victims. A mission to investigate the 
potential victim locations is transferred to the rotorcraft which is sent out to provide a high- 
resolution 3D assessment of the site and to confirm the victim detections using its on-board 
human detector, as shown on Figure 15. The rotorcraft returns from its mission. A very high- 
resolution 3D map of the scan area is transferred, confirming the position of one undetected 
victim. The rotorcraft is sent out again to the victim location to assess the medical state of 
the victim. Analysis of the 3D map, imagery data and the victim location and medical state 
returned by the rotorcraft helps the planning team in setting up a plan to rescue the victims. 
Human rescue team members are sent out to rescue victims which can be evacuated without 
the help of the unmanned tools. The rotorcraft is (manually) equipped with a rescue kit and is 
requested to deliver this rescue kit to a victim which is trapped in the middle of the remains 


Figure 15. Simultaneous aerial victim search operations at different altitudes (source: ICARUS). 
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Figure 16. Rescue kit delivery (source: ICARUS). 


Figure 17. Debris clearance with the large UGV (source: ICARUS). 
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Figure 18. Shoring operation with the large UGV (source: ICARUS). 


of the demolished building, as shown on Figure 16. The large UGV has cleared a pathway 
to the victim, as shown on Figure 17. The human rescue team comes in and evacuates the 
victim. The canine rescue team has located another victim below the rubble. However, due to 
structural instability, the access to the victim is considered too risky for direct human inter- 
vention. The canine rescue team sends the data on the victim location to the RC2 using their 
mobile devices and request the intervention of the large UGV for shoring the access path to 
the victim. Upon receiving this task, the large UGV first heads (remotely controlled) to the 
local command station to pick up few hydraulic/pneumatic struts. The large UGV is in place 
to start the shoring operation, with all tools at hand. The large UGV uses its manipulator arm, 
remotely controlled using an exoskeleton, to place the struts, as shown on Figure 18. The 
struts are activated remotely by a human operator. The large UGV has stabilised the entrance 
towards the sixth victim. Human rescue team members enter and evacuate the victim. 


Evaluation: End users and stakeholders were impressed by the seamless integration and col- 
laboration as a team of ICARUS tools in the B-FAST search and rescue toolkit. The comple- 
mentarity between canine and unmanned aerial search teams was applauded as extremely 
useful for future SAR operations. 


3.6. Victim search and rescue in semi-demolished school building 


Event: The team now focuses its attention on the school building where multiple children 
are reported missing. The building is too unstable for human rescue workers to enter safely. 
Luckily, however, the access to this building is clear. As such, it is decided that the indoor 
rotorcraft and the small unmanned ground vehicle will be sent in simultaneously for assessing 
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Figure 19. Indoor rotorcraft entering the building via broken window (source: ICARUS). 


Figure 20. Unmanned ground vehicle navigating through the corridors of the building (source: ICARUS). 


Figure 21. Indoor rotorcraft and small UGV exploring the school building (source: ICARUS). 
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Figure 22. Civil protection working acting as victim (obviously no real children could be used) (source: ICARUS). 


the structural integrity of the building, giving rescue workers a view of what is happening 
inside and for finding survivors. 


The indoor rotorcraft enters the building by flying through a broken window, as shown on 
Figure 19, while the small unmanned ground vehicle enters through the doors and corridors, 
as shown on Figure 20. Simultaneously, they explore and map the building, as shown on 
Figure 21 both finding and localizing survivors with their infrared and visual sensors, as 
shown on Figure 22. 


The structural scans by the unmanned systems help the planning team in deciding that the 
rescue team can be safely sent into the building to rescue the victims detected and localized 
by the unmanned systems. 


Evaluation: End users were positively surprised about the navigation capabilities in small 
enclosures of both the rotorcraft and the small UGV. Specifically, the capability of the rotor- 
craft of flying through window openings only marginally larger than its proper size gener- 
ated much appraisal. For the indoor operations which happened beyond line of sight, radio 
connectivity was obviously problematic, so the semi-autonomous exploration behaviours of 
both vehicles had to be used. While this led to a slower exploration strategy (notably for 
the ground vehicle) due to safety reasons, the whole planned operation could be completed 
within the time constraints. 


3.7. Victim search and rescue in semi-demolished CBRN (Chemical, biological, 
radiological and nuclear) warehouse 


Event: The mayor reports that there is a two-story warehouse with potentially dangerous 
products near the city centre. As previous attempts to enter the building without using the 
help of unmanned tools have failed and the structure is too unstable for the safe operation of 
humans near the walls, the help of all unmanned tools is requested for entering the building. 


243 


244 Search and Rescue Robotics - From Theory to Practice 


Figure 23. Unmanned tools navigating to CBRN warehouse (source: ICARUS). 


Figure 25. Remote operator using an exoskeleton to operate the manipulator arm of the small UGV (source: ICARUS). 
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Figure 26. Small UGV opening a door with the manipulator arm, remotely operated by the exoskeleton (source: ICARUS). 


The large UGV is remotely controlled up to the building walls, as shown on Figure 23. With 
its manipulator arm, the large UGV deploys the small UGV which it was carrying onto a 
terrace on the first floor of the building, as shown on Figure 24. The small UGV moves to 
the door on the first floor. The exoskeleton is then used by a remote operator to control the 
manipulator arm on the small UGV to open the door, as shown on Figure 25 and Figure 26. 
The small UGV is now inside the building and starts scanning the surroundings, searching for 
victims and dangerous products and inspecting the structural integrity of the building. The 
complete building is explored by the small UGV and two human survivors are detected and 
localized. Meanwhile, the large UGV is standing guard close to the building entrance to act 
as a wireless repeater, ensuring optimal communication to the small UGV inside and the out- 
door rotorcraft is sent to explore the roof of the building to detect possible human survivors. 
The outdoor rotorcraft detects a human survivor on the rooftop. Visual inspection from the 
footage of the UAS also indicates that the emergency exits seem to be blocked, indicating that 
the survivors should be evacuated through the building. Based on the data provided by the 
rotorcraft and the small UGV, the rescue workers decide on the protective equipment to wear 
for entering the building and start the victim evacuation. 


Evaluation: The public was impressed by the capability of the ICARUS command and control 
system to seamlessly manage the simultaneous operation of three unmanned tools (small 
UGV, large UGV and outdoor rotorcraft). Also the exoskeleton received a lot of attention and 
was praised by the spectators by its user-friendliness. Many stakeholders present at the dem- 
onstration knew (some also from their own experience) that opening a door with a robot 
manipulator without direct line of sight for the operator is a notoriously difficult operation 
and they could not believe how seemingly effortless this operation was achieved via the 
exoskeleton. 


3.8. Conclusions of the land demonstration 


Not less than five unmanned assets (large UGV, small UGV, fixed-wing aircraft, outdoor 
rotorcraft and indoor rotorcraft) collaborated during the different land demonstration 
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missions. All robotic assets were interfaced via the common ICARUS command and control 
interface, thereby showcasing the versatility and modularity of the ICARUS interoperability 


concept. 


All victims who were hidden throughout the different missions were located in the end and 
received assistance, and a tight interrelation and even complete integration of ICARUS tools 
within the toolkit of the search and rescue workers was achieved. 


As discussed before, a series of metrics and key performance indicators were defined by end 
users with detailed target performance levels to be attained by the different systems. For none 
of those metrics, a score was noted which formed a problem for global mission success. For 
12% of the capabilities and metrics, the minimum acceptance level as imposed by the end 
users was reached, but the initially imposed target performance was not reached. For no less 
than 33% of the capabilities and metrics, the goal performance level as imposed by the end 
users was reached or even surpassed. Finally, for about half (47%) of the capabilities and met- 
rics, consisting of the majority of the performance bins, a performance level which was indi- 
cated by the end users as “breakthrough-level” prior to the demonstrations was reached. This 
number is an extraordinary achievement, showcasing that the ICARUS project succeeded in 
surpassing by far the expectations of the end users. 
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-As of 26 May, floodwaters 
continue receding. However, 
many rural areas around Samca, 
Brcko, Orasje and Bijeljina are 
still covered with water (as of 25 
May, UN). 

+ According to the report of UN 
Country Team in Bosnia and 
Herzegovina (as of 25 May): 24 
people were killed (number not 
officially confirmed), more than 
2 000 landslides damaged roads 
and homes around the country 
(over 1 000 landslides only in 
Tuzla Canton); many roads were 
still in need of repair on 25 May 
and many areas remained without 
electricity. The affected 
population is estimated to be 
1.5 million people (as of 24 
May, UN, see Table in the map). 

+ Highest needs remain the 
provision of clean drinking water, 
healthcare and epidemics 
prevention and food for the 
displaced. Additionally, the 
clearance of displaced land mines 
and the monitoring of the active 
landslides areas are among the 
highest priorities. 


ECHO, Local Media, 
UN (2, 2) 


‘Sources: 
Hydrometeorological Institutes, 


Location of UCPM assistance 


| Affected areas (As of 22 May, UN OCHA) 
Main roads 


Main rivers 


Floodwaters detected by NASA-MODIS, 
over 24-25 May. 


Floodwaters detected by Copernicus 
Emergency Management Service 
(Observations available for the regions 
of Doboj, Orasje, Brcko, Bijeljina, 
Vlasenica and Zenica! 


Figure 27. Bosnia flood map (source: European Union Joint Research Centre —source: European Union 2014—used with 


permission). 
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4. Real flood relief operation 


4.1. Mission context 


At the end of May and the beginning of June 2014, a catastrophic massive flooding occurred in 
Bosnia and Herzegovina, Croatia and Serbia due to abundant rainfall over the course of a few 
weeks. All countries suffered immense damage. In Bosnia and Herzegovina, where the whole 
northern region and part of the central region were heavily affected, it was estimated that 1.5 
million people were affected (accounting to 39% of the population). Floods and landslides 
were responsible for at least 53 deaths in Bosnia and Herzegovina and Serbia [10]. 


In response to this catastrophe, the EU Civil Protection Mechanism was activated. Twenty-two 
EU member states offered assistance through the mechanism. Figure 27 shows the flood situa- 
tion and the deployment of international response teams, activated via the EU Civil Protection 
Mechanism. However, the relief efforts were hampered by the destroyed infrastructure, bro- 
ken telecommunications, blackouts, etc. [10]. 


Making matters worse, Bosnia and Herzegovina was contaminated with landmines due to the 
war that took place there from 1992 to 1995, and as a result, the country has one of the most 
serious landmine problems in the world. The presence of many explosive remnants of war 
(ERWs) remaining from the Balkan War of the 1990s created a very dangerous situation for 
the relief workers and the local population. The floods, torrents, landslides and land-shifting 
had a destructive impact on the (previously mapped) suspected hazardous areas (SHAs) 
and minefields in Bosnia and Herzegovina. In Bosnia, 831.4 km? of SHAs were flooded, and 
37.48 km? of SHAs in 33 locations were under direct impact of landslides and torrents. Due 
to the floods, the ERW started moving place and the SHAs had to be extended dramatically. 
By the 4th of July 2014, 1018 pieces of unexploded ordnance (UXO), 92 mines and 3 cluster 
bombs were already found, as well as 40,163 pieces of ammunition. Moreover, 80.2 km? of 
new areas that were previously not been suspected of having mines became potentially haz- 
ardous (mainly in the northern part of Bosnia and Herzegovina). The Bosnian Mine Action 
Centre (BHMAC) was immediately deployed and provided data and information about the 
affected regions, the types of influence, the impact intensity, the spatial distribution, as well 
as the priorities [11]. Obviously, the problem of shifting minefields also hampered the provi- 
sion of aid and relief, as well as the clearing of debris, as relief workers had to proceed with 
much care. 


Among many other international Search and Rescue (SAR) teams, the Belgian First Aid and 
Support Team (B-FAST), which is an ICARUS project partner, was deployed in Bosnia to help 
with relief operations. In order to put into practice the research efforts performed within the 
ICARUS project, the Belgian Royal Military Academy decided to send along with the B-FAST 
team a UAV and a trained operator, together with 3D mapping tools, in order to assist the 
teams with tasks such as damage assessment, situational awareness, dike breach detection, 
mapping, aerial inspection and relocalizing the many ERWs that were displaced due to the 
landslides [12]. The computing and data management tools described in Chapters 6, 8 and 9 
were used in order to give the end users access to the data gathered by the unmanned system. 
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This mission fitted perfectly in the framework of the European research projects ICARUS 
[7] and TIRAMISU (on humanitarian demining) [13]. On the terrain, we were deployed in 
assistance to a team of the Bosnian Mine Action Centre in multiple regions of the country in 
order to localize the displaced ERWs. In a first phase, we provided support for urgent actions 
(urgent demining, assessment of status of minefields, etc.) by performing aerial surveys. 


As we wanted to be fully incorporated into the deployment of the multiple international 
rescue teams, we first had a coordination meeting in the capital city of Sarajevo with the 
Bosnian Ministry of Security. After discussion with the Ministry of Security of Bosnia and 
Herzegovina and the national Directorate of Civil Aviation (BHDCA), flight permits for our 
operations were granted up to a flight altitude ceiling at 150 m for the complete Bosnian terri- 
tory. Due to the crisis, and because all application documents for the flight permits were read- 
ily available (as they were already prepared for the previous operations), these flight permits 
were issued within half a day. 


During a 2-week period, we deployed a vertical take-off and landing remotely piloted aircraft 
system at 13 locations in the north and central part of the country. In total, we performed 
about 20 flights within visual line of sight in semi-urban and urban. Two types of operations 
were performed: 


e Manual flights, where end users (demining or rescue teams) indicated interest points they 
wanted to see investigated by the UAV, mainly for damage assessment and visual inspec- 
tion. A trained operator executed the flights themselves. 


Figure 28. Top left: Flooded city of Orasje; top right: UAV used for the operations; bottom left: optimization of the location 
for the B-FAST water pumps and bottom right: broken dam on the Sava river detected by the UAV (source: ICARUS). 
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e Waypoint-based mapping flights, where an area to be mapped was indicated by the end 
users. A flight plan for the UAV was then set up to map this area using an autonomous 
waypoint-based flight. Also for these operations, a trained pilot always supervised the re- 
mote control station. 


A typical flight lasted around 25 to 30 min, enabling us to cover an area of about 1 hectare. 
Multiple mapping missions were performed, gathering from 200 images to a maximum of 500 
images, all with a resolution of 24 megapixels and mapping areas as large as 1 km’. 


4.2. Flood relief operations 


The Belgian B-FAST team was deployed to the city of Orasje (located in the northeast), which 
was one of the cities that was hit most by the floods. The UAV was first deployed here to assist 
the B-FAST team to monitor the water levels and to assess the optimal location to install the 
high-pressure pumps, as shown in Figure 28. The problem with the installation of the water 
pumping system was that water levels were not decreasing after multiple days of pumping, 
due to an undetected dike breach. 


The ICARUS-TIRAMISU UAV was able to locate this broken dam, as shown in Figure 28. 
Expert analysis of the UAV imagery indicated that this dam breach was not caused by natu- 
ral means. As a result, the Bosnian Ministry of Justice has initiated a criminal investigation 
against the individual(s) who caused it and commissioned the ICARUS-TIRAMISU UAV 
image material as evidence. The UAS proved very useful to quickly detect dike breaches and 
to map the area quickly. One of the main challenges was to find a landings spot on dry land, 


Figure 29. Top left: City of Maglaj; top right: re-location of mines due to the landslides; bottom left: detected anti- 
personnel mine (mine moved due to the landslides); bottom right: damage assessment for mapping infrastructure 
damage) (source: ICARUS). 
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as there were virtually no spots of clear and open land suited for take-off and landing. Due to 
these difficult operating conditions, all take-off and landing operations were done via remote 
control by a trained pilot. 


Next to the operations in support of B-FAST, the UAV was also deployed at the request of 
the German Federal Agency for Technical Relief (THW) team and Austrian relief workers 
working at the incident site. These teams asked for the help of our UAV system for damage 
analysis, aerial inspection, improving their situational awareness and for selecting the opti- 
mal location for the installation of the high-pressure water pumps. The Ministry of Security 
and the Federal Civil Protection of Bosnia and Herzegovina also requested for UAV support 
in the region of Kopanice (Southeast of Orasje), where the floodwaters from the Sava River 
broke through the local dams. The floodwaters flowed through the breaches and entirely sub- 
merged the farming lands, and all the people needed to be evacuated. The broken dam area 
was in a mine-suspected region, making this mission especially risky. 


Figure 30. High-resolution orthomosaic of a mine-affected area (source: ICARUS). 


Figure 31. Digital elevation model of a minefield affected by a landslide (source: ICARUS). 
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4.3. Demining support operations 


Another city that was hit hard by the floods and landslides was the city of Maglaj, shown 
in Figure 29. An extra problem in this region was the presence of many ERWs, making the 
deployment and work of the relief teams very dangerous. Therefore, it was decided to deploy 
the UAV system for inspection flights, specifically into areas that the relief teams could not 
easily access due to the mine risks. The UAV was used for aerial assessment and mapping of 
mine-suspected areas and to find indicators of where the minefields were shifted due to the 
floods and landslides. Figure 29 shows such a reallocated minefield due to landslides. The 
data of the UAV was very important in assessing the ground movement due to landslides, 
as shown in Figure 29. From this information, experts could deduce the area where the land- 
mines were moved to, which allowed to drastically reduce the search areas. 


Connection type ~~. Time(min) Theoretical upload speed (Mbit/s) 
High-speed Ethernet 6 5.5 

High-speed Wi-Fi 9 5.38 

Mobile connection using mobile phone as router 67 1 

Local network of the cloud system (Wi-Fi) 0.79 70 


Table 1. Data upload speed for the cloud system (dataset: 500MB). 


Figure 30 and Figure 31 show the initial post-processing results of the Dolac region in central 
Bosnia and Herzegovina. The UAV was used in this region for providing orthophotos, 3D maps 
and Digital Elevation Models of the environment to analyse the effects of the landslides on 
mines and ERWs. These results were used as initial models by BHMAC for spatial estimation of 
new hazardous risks caused by the movement of unexploded ordnance (UXO) and landmines 
to areas that were not mine-infested before the disaster. Combining the data from the UAV (3D 
Digital Terrain Models) with pre-existing data (mine risk maps from the Mine Action Centres 
and satellite imaging), it was possible for experts to predict the movement of the landmines and 
to create updated maps of mine-affected areas and mine risk map. To give an indication of the 
scale of the problem: some mines were found up to 23 km from their original location. If mines 
can move over such long distances, this means that the search area is enormous and it also 
means that area reduction techniques, such as the use of UAVs, combined with 3D mapping 
tools predicting the ERW movements and limiting the search area, have a major impact in the 
disaster-response operation. More detailed information about this mission can be found in [14]. 


4.4. Data fusion and sharing 


While the UAV tool provided a wealth of 3D information, one issue was that this valuable 
data had to be distributed quickly and securely to widely dispersed stakeholders (Bosnia and 
Herzegovina Ministry of Security, demining experts within the Mine Action Centre and deployed 
international search and rescue teams). An important constraint for doing this was that these 
stakeholders in general did not dispose of advanced 3D-viewing tools. As a result, we decided 
to upload the gathered datasets to a cloud system (see Chapter 9). During this field operation, 
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this was done by directly copying the datasets to the cloud system [15]. We thereby tested the 
upload speed for a dataset of 500 MB, representing an average size of a dataset acquired by one 
sensor system (point clouds from a laser scanner or images from the digital camera of the UAS) 
for a single team area of operation (roughly 200 x 200 m). The results are shown in Table 1: 


From our point of view, the best configuration is the local network of the cloud system, as it 
shows limited upload times inside the local cloud created by the server. After uploading and 
processing the datasets, they are immediately available to the end users for analysis and help- 
ing in the decision-making processes. Data processing requires an operator to combine different 
types of datasets. The operator is also responsible for choosing the correct parameters for the 
different processing step and also evaluating the results. To facilitate the work of the operator, 
a set of recommended parameters were provided with a software module. The processed data 
were provided to the end users via a set of dedicated tools, focusing on efficient data visualiza- 
tion. This entire process is also secure (which is important, as UAV data is in general considered 
sensitive by governments), as the data itself is not passed directly, but in the form of renders. 
This also requires also only a very small amount of computation power on the local hardware 
of the end users. The provided server system is scalable, as it allows easy integration of extra 
software. 


Using this tool, a demining expert from the Bosnia and Herzegovina Mine Action Centre was 
able to browse through the UAS data sets and make use of his prior experience for identify- 
ing indicators of mine presence under the destructive impact of the landslides and floods. He 
could define new maps of mine-affected areas and create updated mine risk maps. This new 
information of the mine action situation could be easily shared via the cloud with other end 
users (mainly SAR teams in this case) in order to enhance their safety on the terrain. 


4.5. Conclusions of the flood response operation 


Overall, the UAV mission proved to be a real success, based on the feedback provided by the 
end users. Some of their comments are as follows: 


The first flight with the UAV was an assessment inspection of the flooding area where the B-EAST water 
pumps were prepared. To get relevant information of this area from water and land would have cost us 
about 3 days. With the UAV we were able to provide even better results within 2 hours. With the second 
flight, we obtained a 3D model of the area on which the B-FAST could determine the natural flow of the 
water. This information was not even attainable from the ground. (B-EAST Team leader). 


The results obtained by the UAV have been of the utmost importance during the response period, and 
also for post processing and investigation of future activities. (High ranked representative of the Min- 
istry of Security Bosnia and Herzegovina). 


The rapid mapping activities and the results we obtained from the UAV mission were crucial for dam- 
age assessment, and for relocalizing the many explosive remnants of war that were displaced due to 
the landslides and flooding. In that situation, we did not risk putting humans in the danger zones. 


(Technical operation officer of BHMAC). 


This valuable feedback from the end users during a real relief mission clearly shows that the 
mission really had an impact on the terrain and that the use of novel technological tools devel- 
oped within the ICARUS project in real search and rescue missions provides an added value. 
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5. General conclusions and acknowledgements 


This concluding chapter has shown how the unmanned tools for search and rescue, devel- 
oped within the framework of the ICARUS project, have been operationally validated during 
large-scale simulated demonstrations and even during real-life interventions. A main focus 
point during all these validation events was a very tight integration of the ICARUS tools 
into the operational toolbox of real search and rescue workers and the integration into their 
standard operating procedures, in order to not only validate the technical capabilities of the 
robotic systems which were developed, but also validate the deployment, command and con- 
trol, training and support tools which were developed for the end users. 


The validation was performed by putting the ICARUS tools in the hands of end users and 
letting them learn to use the systems and letting them evaluate their experiences. The out- 
come of all these validation trials was extremely positive. Testimony to this statement is that 
multiple ICARUS end users have decided —after witnessing the positive effects of the use of 
ICARUS tools on the terrain—to start the acquisition of their own unmanned systems, and 
then mostly unmanned aerial vehicles that show the most short-term potential for improv- 
ing the effectiveness of search and rescue operations. Of course, we must not be blind to the 
still-existing bottlenecks for the integration of unmanned SAR tools in real rescue operations. 
For aerial systems, access to airspace is subject to national legislation which is very different 
from one country to another, which can seriously restrict the use of these systems. For ter- 
restrial systems, mobility on rough terrain and stair-climbing ability are still unsolved issues. 
Likewise, the unmanned maritime systems still have to prove their effectiveness in very rough 
sea conditions. 


All the presented validation trials would not have been possible without the kind support of 
many end-user organisations which contributed to the definition of the validation protocols. 
Specifically, we would like to thank the Portuguese Navy and Belgian Defence for the sup- 
port received during the marine and land demonstration events. We also wholeheartedly want 
to thank the Belgian First Aid and Support Team and the Bosnian Mine Action Centre who 
took up the challenge to allow the integration of novel technological tools developed within 
a research project inside the real relief mission for the Bosnian floods. This must have been a 
gamble with uncertain outcome for them at the time of the decision, but it turned out extremely 
positive for all partners and led to a better mutual understanding between researchers/platform 
developers and the end-user community. This tight intertwining of the research community 
and the end-user community has been the main focus of the ICARUS project, such that solu- 
tions could be developed which make a real difference on the terrain. Within this chapter, we 
have clearly shown that such an impact could be made for all of the ICARUS tools discussed 
within Chapters 3-9. 
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